Apparatus with Adaptive Microphone Configuration Based on Surface Proximity, Surface Type and Motion

ABSTRACT

An apparatus includes a group of microphones and a surface compensator that is operatively coupled to switch logic and to a signal conditioner that provides a control channel to voice recognition logic. The surface compensator may detect surfaces in proximity to the apparatus as well as the surface&#39;s acoustic reflectivity or acoustic absorptivity and may accordingly configure the group of microphones including selecting appropriate signal conditioning and beamforming based on the surface acoustic reflectivity or acoustic absorptivity and the orientation of the apparatus. Voice recognition performance is thus improved when microphones are impeded or occluded by proximate surfaces. A group of sensors of the apparatus is used by the surface compensator to detect surfaces and surface type, and to determine apparatus orientation and motion.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 61/827,569, filed May 25, 2013, entitled “VOICERECOGNITION SYSTEM WITH ADAPTIVE ADJUSTMENT BASED ON SURFACE PROXIMITY,SURFACE TYPE AND MOTION” which is incorporated in its entirety herein,and further claims priority to U.S. Provisional Patent Application No.61/798,097, filed Mar. 15, 2013, entitled “VOICE RECOGNITION FOR AMOBILE DEVICE,” and further claims priority to U.S. Provisional Pat.App. No. 61/776,793, filed Mar. 12, 2013, entitled “VOICE RECOGNITIONFOR A MOBILE DEVICE,” all of which are assigned to the same assignee asthe present application, and all of which are hereby incorporated byreference herein in their entirety.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to mobile devices and moreparticularly to voice processing and to voice recognition systems forsuch mobile devices.

BACKGROUND

Mobile devices such as, but not limited to, mobile phones, smart phones,personal digital assistants (PDAs), tablets, laptops or other electronicdevices, etc., increasingly include voice recognition systems to providehands free voice control of the devices. Although voice recognitiontechnologies have been improving, accurate voice recognition remains atechnical challenge.

A particular challenge when implementing voice recognition systems onmobile devices is that, as the mobile device moves or is positioned incertain ways, the acoustic environment of the mobile device changesaccordingly thereby changing the sound perceived by the mobile device'svoice recognition system. Voice sound that may be recognized by thevoice recognition system under one acoustic environment may beunrecognizable under certain changed conditions due to mobile devicemotion or positioning.

More specifically, the mobile device acoustic environment impacts theoperation of signal processing components such as microphone arrays,noise suppressors, echo cancellation systems and signal conditioningthat is used to improve voice recognition performance. When a mobiledevice is placed on or near a table or other surface, the result is achange of the transfer function between the mobile device microphones,the relative signal strength and phase of the microphone signals, andcreation of additional reflections, all of which negatively impact theoperation of the signal processing components, rendering themsub-optimal. For example, where a microphone array is used, changes inthe acoustic environment results in undesirable changes in the arraydirectivity, and increases pick-up of undesired sound sources.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram of an apparatus having a voicerecognition system in accordance with the embodiments.

FIG. 2 is a flow chart showing a method of operation of an apparatuswith a surface compensator in accordance with various embodiments.

FIG. 3 is a flow chart providing details of one operation of the methodshown in FIG. 2, in accordance with an embodiment.

FIG. 4 is a flow chart providing details of another operation of themethod shown in FIG. 2 in accordance with an embodiment.

FIG. 5 is a flow chart providing details of another operation of themethod shown in FIG. 2 in accordance with an embodiment.

FIG. 6 is a flow chart showing a method of operation of an apparatuswith a surface compensator in accordance with various embodiments.

DETAILED DESCRIPTION

The present disclosure provides an apparatus and methods of operationthat improve voice recognition performance of a voice recognition systemin a mobile device. The voice recognition performance is improved in thevarious embodiments by providing adaptive adjustment of microphoneconfiguration including switching off occluded microphones or adjustmentof signal conditioning and beamforming parameters based on surfaceproximity to the mobile device, the type of surface in proximity, and,in some embodiments, the mobile device motion. The mobile device detectsproximal surfaces with respect to the mobile device's orientation,classifies the surfaces based on surface reflectivity or absorptivity,and accordingly configures the microphones and/or adjusts the variousparameters. Mobile device sensors are used to obtain a signature thatmay be correlated to the surface type and acoustic reflectivity oracoustic absorptivity characteristics using predetermined signaturesstored in memory.

The present disclosure provides a method of operation that includesdetecting a surface in proximity to a mobile device using sensor data,and determining whether the surface is acoustically reflective oracoustically absorptive. The method of operation may further includecompensating for the surface acoustic reflectivity or absorptivity bycontrolling a configurable group of microphones of the mobile device.The present disclosure provides another method of operation thatincludes determining that a first microphone, of a group of microphonesof a mobile device, is impeded by a surface that is in proximity to themobile device, and turning off the first microphone. The method mayinclude turning on a second microphone that is determined not to beimpeded by the surface. The method may include detecting the surfacethat is in proximity to the mobile device, and using sensor dataobtained from at least one sensor of the mobile device to determineacoustic reflectivity or acoustic absorptivity of the surface.

The method may further include determining orientation of the mobiledevice, and determining that the first microphone is impeded by thesurface using the orientation and the first microphone's known positionin the mobile device. The method may further include selecting signalconditioning parameters based on the acoustic reflectivity or acousticabsorptivity of the surface. Selection of signal conditioning parametersbased on the acoustic reflectivity or acoustic absorptivity of thesurface may be accomplished in some embodiments by selecting signalconditioning parameters from a memory table correlating a plurality ofsets of signal conditioning parameters to surface types forpredetermined conditions. De-reverberation may also be applied to anoutput of the second microphone when the surface acoustic reflectivityor acoustic absorptivity characteristic is determined to be reflective.

The method of operation may also include turning on a third microphonethat is determined not to be impeded by the surface and beamforming theoutputs of the second microphone and the third microphone to create avoice recognition virtual microphone.

In some embodiments, a surface that is in proximity to the mobile devicemay be detected by actuating a vibrator of the mobile device andobtaining data from an accelerometer of the mobile device where the dataexhibits a signature of the surface's surface type.

The present disclosure provides one embodiment of an apparatus thatincludes a configurable group of microphones with each microphone of thegroup operative to be turned on or off, or have gain, attenuation orfiltering applied to a corresponding microphone output in response tocontrol signals. The apparatus also includes a signal conditioner thatis operatively coupled to the configurable group of microphones and asurface compensator operatively coupled to the configurable group ofmicrophones and to the signal conditioner. The surface compensator isoperative to detect a surface in proximity to the apparatus anddetermine a surface acoustic reflectivity or acoustic absorptivity. Thesurface compensator may be further operative to control the configurablegroup of microphones or the signal conditioner to compensate for thesurface acoustic reflectivity or acoustic absorptivity. The presentdisclosure also provides an apparatus in another embodiment thatincludes a group of microphones or microphone array, and switch logicoperatively coupled to the group of microphones. The switch logic isoperative to switch each microphone of the group of microphones eitheron or off in response to a first control signal. A signal conditioner isoperatively coupled to the switch logic and to the group of microphones,and is operative to receive outputs from each microphone of the group ofmicrophones. Voice recognition logic is operatively coupled to thesignal conditioner. A surface compensator is operatively coupled to theswitch logic and to the signal conditioner, and is operative todetermine that a first microphone, of the group of microphones, isimpeded by a surface that is in proximity to the apparatus andaccordingly turn off the first microphone by sending the first controlsignal to the switch logic.

The apparatus may further include a plurality of sensors, operativelycoupled to the surface compensator. In one embodiment, the apparatusincludes a vibrator and an accelerometer, both operatively coupled tothe surface compensator, where the accelerometer is one sensor of theplurality of sensors. The surface compensator is operative to detect asurface in proximity of the apparatus by actuating the vibrator andobtaining data from the accelerometer where the data exhibits asignature of the surface's surface type. The surface compensator is alsooperative to select signal conditioning parameters based on thesignature of the surface's surface type and adjust the signalconditioner with the selected signal conditioning parameters.

In some embodiments, the apparatus includes changeover switch logic,operatively coupled to the switch logic, and a beamformer, operativelycoupled to the changeover switch logic and to the signal conditioner.The beamformer is operative to receive and beamform microphone outputsfrom the changeover switch logic to create a virtual microphone and toprovide at least one voice recognition channel from the virtualmicrophone to the signal conditioner. The surface compensator isoperatively coupled to changeover switch logic, and is operative to senda second control signal to the changeover switch logic to control thechangeover switch logic to switch each microphone of the group ofmicrophones either to the beamformer or to the signal conditioner.

In some embodiments the apparatus includes a 3D gyroscope where the 3Dgyroscope is one of the plurality of sensors. The surface compensatormay determine orientation of the apparatus by obtaining 3D gyroscopedata and may determine that the first microphone, of the group ofmicrophones, is impeded by the surface based on the orientation of theapparatus and the first microphone's know position in the apparatus.

In some embodiments, the signal conditioner may include a noisesuppressor and a de-reverberator. Also in some embodiments, theapparatus may include global positioning system logic, operativelycoupled to the surface compensator. The surface compensator may receiveglobal positioning system data from the global positioning system logicand determine when the apparatus is in motion in a vehicle based on theglobal positioning system data. The apparatus may further includenon-volatile, non-transitory memory, operatively coupled to the surfacecompensator. The surface compensator may access the memory to comparethe observed signature of the surface's surface type to a table ofstored predetermined signatures and select the signal conditionparameters from the table based on a matching predetermined signature.

Turning now to the drawings, FIG. 1 illustrates a schematic blockdiagram of a mobile device 100, which is an example apparatus inaccordance with the embodiments. The mobile device 100 is an exampleapparatus that has the features and advantages of, among other things, asurface compensator 200. It is to be understood that FIG. 1 is limitedto showing only those components useful to describe the features andadvantages of the various embodiments, and to describe how to make anduse the various embodiments to those of ordinary skill. It is thereforeto be understood that various other components, circuitry, and devicesetc. may be present in order to implement an apparatus such as mobiledevice 100 and that those various other components, circuitry, devices,etc., are understood to be present by those of ordinary skill. Forexample, the mobile device 100 may include inputs for receiving powerfrom a power source, a power distribution bus, a battery, communicationbuses for sending control signals or other information betweenoperatively coupled components, etc. Thus it is to be understood thatsuch various other components, circuitry, or devices are omitted for thepurpose of clarity.

The various components, circuitry, devices etc. described with respectto the figures including, but not limited to, those described using theterm “logic,” may be implemented in various ways such as by softwareand/or firmware executing on one or more programmable processors such asa central processing unit (CPU) or the like, or by ASICs, DSPs, FPGAs,hardwired circuitry (logic circuitry), or any combinations thereof.

The terminology “operatively coupled” as used herein refers to couplingthat enables operational and/or functional communication andrelationships between the various components, circuitry, devices etc.described as being operatively coupled and may include any interveningitems (i.e. buses, connectors, other components, circuitry, devicesetc.) used to enable such communication such as, for example, datacommunication buses or any other intervening items that one of ordinaryskill would understand to be present. Also, it is to be understood thatother intervening items may be present between “operatively coupled”items even though such other intervening items are not necessary to thefunctional communication facilitated by the operative coupling. Forexample, a data communication bus may be present in various embodimentsand may provide data to several items along a pathway along which two ormore items are operatively coupled, etc. Such operative coupling isshown generally in FIG. 1 described herein.

In FIG. 1 the mobile device 100 is an example apparatus in accordancewith the embodiments and includes a voice recognition system. The mobiledevice 100 may be any of various mobile devices such as, but not limitedto, a video camera, smartphone, tablet computer, laptop, audio recorder,or some other electronic device, etc. In the mobile device 100, a groupof microphones 110 provide microphone outputs that are operativelycoupled to microphone configuration logic 300. In some embodiments, themicrophone configuration logic 300 may include switch logic 120, frontend processing 130 and changeover switch logic 140. The microphoneconfiguration logic 300 together with the group of microphones 110 formsa configurable group of microphones that may be each individually turnedon or off, or have gain, attenuation or filtering applied to acorresponding microphone by the front end processing 130. In otherwords, the microphone configuration logic 300 is controllable byappropriate control signals. Although the example of FIG. 1 shows threemicrophones, the embodiments are not limited to three microphones andany number of microphones may be used in the embodiments. Eachmicrophone M1, M2 and M3 of the group of microphones 110, is operativelycoupled to a corresponding switch S11, S12 and S13 of switch logic 120.The switches S11, S12 and S13 are operative to turn their respectivemicrophones on or off in response to a control signal received from asurface compensator 200. That is, the switches S11, S12 and S13 operateas single pole, single throw (SPST) on-off switches. The switches may beimplemented in any suitable manner such as by tri-state logic (i.e. atri-state buffer as an SPST switch, etc.) or some other implementationas understood by those of ordinary skill.

The surface compensator 200 is operatively coupled to the switch logic120 by, for example, an internal communication bus of the mobile device100. The surface compensator 200 provides control signals 143 to theswitch logic 120 and may turn any one of the microphones M1, M2 and M3on or off in any combination by sending the appropriate control signal143 to the appropriate switch S11, S12 or S13, respectively.

The outputs of switch logic 120 are operatively coupled to front endprocessing 130, which may include, but is not limited to, signalamplification, analog-to-digital conversion/digital audio sampling, echocancellation, etc., which may be applied to the physical microphone M1,M2, M3 outputs prior to the beamformer 150 or signal conditioner 160. Insome embodiments, the front end processing 130 may receive a controlsignal 145 from the surface compensator 200 to adjust parameters suchas, for example, amplifier gain.

The front end processing 130 is further operatively coupled tochangeover switch logic 140. Changeover switch logic 140 is operativelycoupled to the surface compensator to receive control signal 147, andincludes a switch operatively coupled and corresponding to eachmicrophone output, for example switches S21, S22 and S23. The switchesS21, S22 and S23 are configured as a single-pole, double throw (SPDT)changeover switches in that each switch is operative to switch itsrespective microphone output to the beamformer 150 or directly to thesignal conditioner 160. In other words, the changeover switch logic 140is controlled by the surface compensator 200 to turn beamforming on oroff. In other embodiments, the beamformer 150 may be turned on or off bycontrol signal 153 from the microphone configuration controller 210.Microphones may be effectively turned on or off by the beamformer 150 byincreasing gain on microphone inputs to turn microphones on and byadjusting gain to zero to turn microphones off That is, in someembodiments the beamformer bypass operation of the changeover switchlogic 140 may be performed by the beamformer 150 and the changeoverswitch logic 140 and/or switch logic 120 may not be used or may not bepresent in the embodiment. Instead, in some embodiments, the microphoneconfiguration controller 210 may control the beamformer 150 by controlsignal 153 to implement the operations of the microphone configurationlogic 300 switches.

The beamformer 150 is an adjustable beamformer and receives audiosignals from the microphone M1, M2 and M3 outputs and performsbeamforming to create various virtual microphone audio channels thatinclude at least one voice recognition control channel 125. In someembodiments, the beamformer 150 may provide other virtual microphoneaudio channels such as audio channels 121 and 123, which may be, forexample, virtual microphone voice communication channels on a mobilephone, virtual microphone voice recording channels on a video recorder,or some other type of virtual microphone audio channel. In otherembodiments, multiple voice recognition channels may be created bybeamformer 150 in addition to voice recognition control channel 125.

The beamformer 150 creates the virtual audio channels by applyingappropriate filtering algorithms and using a group of adjustable filters151 which, in some embodiments, may be adaptive filters. In one example,sound received by microphones M1, M2 and M3 may be combined andappropriately filtered to create a single “virtual microphone” thatreceives sound from a given direction. As mentioned above, the group ofmicrophones 110 may include several microphones (i.e. a microphonearray) such that one or more virtual microphones may be created usingseveral physical microphone outputs and applying beamforming techniques.As a few examples, the created virtual microphones may be used to focuson sound in any given direction or to pan a room to search for specificvoices, sounds, etc. Therefore, the beamformer 150 may be used to createone or more virtual microphones to pick up sound coming from thedirection of the controlling user while another virtual microphone maypick up sound in a direction where another speaker is located such as,for example, in a conference call situation where two speakers arelocated in different areas of a room.

The example beamformer 150 includes various filters such as filter 151which provide the various other audio channels 121 and 123 and the voicerecognition control channel 125. As shown in FIG. 1, the beamformer 150may have any number of filters. Likewise, the number of virtual audiochannel outputs is not limited. The filters 151 may be finite impulseresponse (FIR) filters in some embodiments and may also be adaptive FIRfilters. Additionally, in some embodiments, outputs from one or morefilters may be combined, or summed, to create a virtual audio channel.That is, in accordance with the embodiments, any appropriate filterconfiguration may be implemented within the beamformer 150.

The beamformer 150 is operatively coupled to a signal conditioner 160and provides the virtual microphone audio channels as inputs. It is tobe understood however that the signal conditioner 160 could also belocated prior to the beamformer 150 and therefore may be operativelycoupled to the microphone outputs at changeover switch logic 140 orelsewhere such that the signal conditioner 160 may provide conditionedmicrophone outputs to the beamformer 150 in some embodiments. In theexample embodiment of FIG. 1, the various audio channels 121 and 123,and voice recognition control channel 125 are input to the signalconditioner 160. The signal conditioner 160 is also operatively coupledto changeover switch logic 140, and may receive microphone inputs 149directly when the changeover switch logic 140 is switched to bypass thebeamformer 150. The signal conditioner 160 is also operatively coupledto the voice command recognition logic 101 to provide a conditionedcontrol channel 129 as an input to the voice command recognition logic101. The voice command recognition logic 101 is operatively coupled tothe system control 103, which may be any type of voice controllablesystem control such as, but not limited to, a voice controlled dialer ofa mobile phone, a video system control, an application control of amobile phone, smartphone, tablet, laptop, etc., or any other type ofvoice controllable system control. The voice command recognition logic101 send a control signal 131 to the system control 103 in response todetected command words or command phrases received on the conditionedvoice recognition control channel 129. The signal conditioner 160 mayprovide a conditioned output, corresponding to each signal conditioner160 input. For example, signal conditioner 160 may provide conditionedaudio channels 127 to some other mobile device 100 system such as atransceiver. The transceiver may be a wireless transceiver using any ofvarious wireless technologies and may implement a full duplextelecommunication channel. The conditioned voice recognition controlchannel 129 is input to the voice command recognition logic 101 as noteabove.

The surface compensator 200 is operatively coupled to the signalconditioner 160 to provide a control signal 139. The surface compensator200 may send the control signal 139 when the surface compensator 200detects a reflective or absorptive surface in proximity to the mobiledevice 100. In response to detection of a surface, and determination ofa surface type, the surface compensator 200 may send the control signal139 to the signal conditioner 160 and may control any of various audioattributes related to various audio enhancement functions in order tocompensate for the proximate surface. The signal conditioner 160 mayinclude various audio enhancement functions such as, but not limited to,de-reverberator 161, automatic gain control (AGC) 163, or noisesuppressor 165, etc.

The surface compensator 200 is also operatively coupled to thebeamformer 150 to provide a control signal 153, which may also be sentin response to detection of a surface, and determination of a surfacetype. For example, the control signal 153 may control attributes of thefilters 151 to change the configuration of virtual microphones orredirect the virtual microphone beams, etc., or to adjust some otheraudio attribute. By controlling the filters 151, the beamformer 150 maybeamform the sound received by each of the microphones M1, M2 and M3, inany combination, to create virtual microphones and particularly voicerecognition control channel 125. That is, in accordance with theembodiments, sound received by various physical microphones may becombined in various ways to form virtual microphones that may bereconfigured in response to a microphone impeding surface detected bythe surface compensator 200.

For example, the surface compensator 200 may configure virtualmicrophones to compensate for an impeding proximal surface by focusingon a given direction, amplifying a specific voice or sound, panning, oretc. The surface compensator 200 may include a microphone configurationcontroller 210 that is operative to compensate for surfaces impeding anyone or more microphones of the group of microphones 110 by turningmicrophones on or off and by controlling audio attributes such as, butnot limited to, audio sensitivity, panning in a given direction,focusing on a specified direction, blocking a specified direction, orreducing background noise, etc.

The voice recognition control channel 125 may be created usingbeamforming where the virtual microphone is configured to pick up soundfrom a certain position with respect to the mobile device 100. Moreparticularly, a virtual microphone may be created that is directedtowards the operator of mobile device 100 and that avoids an impedingsurface that impedes one or more physical microphones of the mobiledevice 100. Command word utterances are detected on the voicerecognition control channel 125 by the voice command recognition logic101 and acted upon appropriately.

In one example embodiment, the microphone configuration controller 210of the surface compensator 200 controls the beamformer 150 among otherthings. The example microphone configuration controller 210 may includean internal memory table, or may access a memory table 106 stored insystem memory 105, which is operatively coupled to the surfacecompensator 200 by connection 137. The memory table 106 may storepreconfigured beam patterns for predetermined virtual microphoneconfigurations. The preconfigured beam patterns may be stored by storingfilter coefficients or other filter parameters that may be used by thebeamformer 150 and applied to the group of filters 151 to create thedesired audio beam patterns. The microphone configuration controller 210is operatively coupled to the beamformer 150 to send a control signal153 to adjust the beamformer 150 settings. The control signal 153 mayinclude settings obtained from the memory table 106 or may provide apointer to a memory table 106 location that may be accessed by thebeamformer 150 to adjust settings.

The system memory 105 is a non-volatile, non-transitory memory, and maybe operatively coupled to the signal conditioner by connection 133 andmay also be accessible by other components of the mobile device 100 forvarious setting, stored applications, etc. In embodiments where thesystem memory 105 is used to store the memory table 106, the surfacecompensator 200 is operatively coupled to the system memory 105 to senddata to, and receive data from, the system memory 105 over connection137 which is a bi-directional memory interface.

The microphone configuration controller 210 of the surface compensator200 may also control the signal conditioner 160 by using the memorytable 106. That is, the memory table 106 may store various audioparameters for predetermined audio attributes. The stored audioparameters may include noise suppression algorithms that may be selectedfor a noise suppression component of signal conditioner 160 or someother audio related algorithms to achieve the desired audio attributesto overcome voice recognition issues due to motion or an impedingsurface.

It is to be understood that the various “control signals” describedherein with respect to various components such as, but not limited to,the surface compensator 200, switch logic 120, changeover switch logic140, front end processing 130, beamformer 150, signal conditioner 160,etc., may be implemented in various ways such as using APIs between thevarious components. Additionally, operations involving the memory table106 may be implemented using pointers where the components access thememory table 106 as directed by the microphone configuration controller210. In other words, control signal 153 and control signal 139 may beimplemented such that the beamformer 150 and the signal conditioner 160obtain control information such as, but not limited to, audio attributeparameters, filter coefficients etc., from system memory 105 and thatsuch implementations are contemplated embodiments and in accordance withthe embodiments and examples described herein.

The surface compensator 200 may include surface detection andclassification logic 220 and device orientation and movement logic 230,both of which are operatively coupled to each other and to themicrophone configuration controller 210 by communication bus 135 whichmay be internal to the surface compensator 200. The communication bus135 may be implemented as application programming interfaces (APIs)between the surface compensator 200 components in some embodiments. Themobile device 100 includes a group of sensors that are all operativelycoupled to the surface compensator 200 by a communication bus 141. Thegroup of sensors may include, but is not limited to, photosensor 109,photosensor 111 (such as cameras), proximity sensor 113, 3D gyroscope115, vibrator 117 and accelerometer 119. The surface compensator 200 mayalso be operatively coupled to global positioning system (GPS) logic 107by connection 136, to receive location data and speed information whenthe mobile device 100 is in motion such as when it is located in avehicle.

In accordance with the embodiments, the surface compensator 200 uses thegroup of sensors to detect a surface such as when a mobile device isplaced on a table. The surface compensator 200 is operative to enableone or more microphones of the group of microphones 110 as appropriateto avoid the impeding surface. The surface compensator 200 may also usemobile device 100 orientation to select an appropriate microphone. Forexample, microphones M1 and M2 may be located at top and bottom edges ofa front side the mobile device 100 and microphone M3 may be locatedsomewhere on a rear side of the mobile device 100. The deviceorientation and movement logic 230 of surface compensator 200 uses datafrom accelerometer 119 or GPS logic 107 to determine if the mobiledevice 107 is moving or is stationary such as when the mobile device 100is on a table top. More particularly, the device orientation andmovement logic 230 may use the accelerometer 119 or gyroscope 115 todetermine the orientation of the mobile device 100. For example, if themobile device 100 is not determined to be moving and is in anorientation relatively perpendicular (with respect to the ground), thedevice orientation and movement logic 230 may determine that the mobiledevice 100 is stationary and may have been placed on a surface.

If the device is determined to have been placed on or near a surface,the device orientation and movement logic 230 may send a trigger to thesurface detection and classification logic 220 to check for surfaces.However, in other embodiments, the surface detection and classificationlogic 220 may independently perform checks for nearby surfaces byobtaining data from the group of sensors. For example, photosensor 109,photosensor 111 or proximity sensor 113 may provide an indication tosurface detection and classification logic 220 of a nearby surface. Thesurface detection and classification logic 220 actuates the vibrator 117and performs a closed-loop measurement by obtaining feedback from theaccelerometer 119 or the gyroscope 115 or both. The information obtainedis used to determine if a surface is present, such as when the mobiledevice 100 is placed on a table, chair cushion, car seat, next to awall, etc., and to determine a surface classification such as whetherthe surface is reflective or absorptive. The orientation of the mobiledevice 100 may be used by the surface compensator 200 in conjunctionwith the surface classification information to determine how toconfigure the microphones to compensate for the surface's acousticreflectivity or acoustic absorptivity.

For example, the device orientation and movement logic 230 will informthe microphone configuration controller 210 of the mobile device 100orientation, and the surface detection and classification logic 220 willprovide the surface type (i.e. surface classification; reflective,absorptive, etc.). One or more microphones may be impeded by a surfacedue to surface reflectivity or surface absorptivity (i.e. occlusion)based on the surface type and the mobile device orientation with respectto the surface. For example, if the mobile device is found to beoriented such that microphone M3 is facing downward, the microphoneconfiguration controller can open switch S13 to turn off microphone M3since it may be impeded by a surface. Alternatively, the microphoneconfiguration controller 210 may send control signal 153 to beamformer150 and apply beamforming to one or more of the microphone outputs toavoid interference from the surface. Other beamforming configurationsmay be used when the mobile device 100 is determined to be, or is mostlikely to be, in the user's hand. The microphone configurationcontroller 210 will also control the signal conditioner 160 to adjustvarious audio attributes. For example, if a surface is found to bereflective, the microphone configuration controller 210 may send controlsignal 139 to the signal conditioner 160 to invoke the de-reverberator161. Thus the surface compensator 200, among other features, detects asurface in proximity to the mobile device 100 and determines an acousticreflectivity or acoustic absorptivity of the surface by using sensordata obtained from the group of sensors operatively coupled bycommunication bus 141. The surface compensator 200 compensates for thesurface acoustic reflectivity or acoustic absorptivity by controllingthe configurable group of microphones, controlling the signalconditioner 160, controlling the beamformer 150, or by combinations ofeach of these. Compensating for the proximate surface by the surfacecompensator 200 may include, among other things, inferring a time delayvia the speed of sound and/or estimating the relative strength of theinitial acoustic reflection from that surface via estimates of thesurface reflectivity or absorptivity. This information may be used insome embodiments to configure the parameters of beamformer 150 or ofsignal conditioner 160 modules such as the de-reverberator 161.Additionally, such information along with the known 3D locations of themicrophones can be used by the microphone configuration controller 210and beamformer 150 to determine a useful orientation for a virtualmicrophone and configure the beamform pattern accordingly. In someembodiments, compensating for the proximate surface may also includeusing beamforming to avoid acoustic inference by the surface (i.e.reflected sound waves or sound absorption) or by adjusting signalconditioning such as de-reverberation, gain, etc. to accommodate forreflections or for signal attenuation due to absorption.

Various methods of operation of the surface compensator 200 may beunderstood in view of the flowcharts of FIG. 2 through FIG. 6. Beginningwith FIG. 2, a method of operation begins in operation block 201 inwhich the surface compensator 200 determines the motion status of themobile device 100. In operation block 203 the surface compensator 200determines the orientation of the mobile device 100 such as, forexample, whether the mobile device 100 is oriented perpendicular to theground or horizontal with respect to the ground such as when it isplaced upon a flat surface such as a tabletop or chair cushion, etc. Inoperation block 205, the surface compensator 200 determines if themobile device 100 actually is located on or near a surface. If themobile device 100 is actually located on or near a surface, the surfacecompensator 200 will determine the surface type in operation block 207.The surface type is a surface classification, such as whether thesurface is reflective or absorptive, which may be determined by a sensordata signature obtained by the surface compensator 200. In operationblock 207, the surface compensator 200 will configure the group ofmicrophones 110 and will apply appropriate signal conditioning, whichmay include beamforming, according to the determined motion status,orientation, and surface type. The method of operation then ends asshown.

The flowchart of FIG. 3 provides details of a method of operation thatoccurs within operation block 201 of FIG. 2. That is, in someembodiments as shown in operation block 211, the surface compensator 200may obtain location data such as, but not limited to, GPS data from GPSlogic 107. The GPS data is processed by the surface compensator 200 todetermine the location of the mobile device 100 and if the mobile device100 is in a moving vehicle. In some situations, GPS data may not beavailable such as when the mobile device 100 is unable to access the GPSsignal. In those situations, as shown in operation block 213, surfacecompensator 200 may obtain accelerometer data from the accelerometer119. In operation block 215, the surface compensator 200 may determinemotion status of mobile device 100 based on the obtained location dataif it was available, accelerometer data, or both. The motion statusdetermination is made by the device orientation and movement logic 230of the surface compensator 200. The method of operation in the flowchart of FIG. 3 then ends and, in the flowchart of FIG. 2, the method ofoperation proceeds to determine the mobile device 100 orientation asshown in operation block 203. Returning briefly to operation block 203of FIG. 2, the device orientation and movement logic 230 may assess dataobtained from the 3D gyroscope 115 to determine if the mobile device 100is oriented horizontally or at some acute angle with respect to theground such that it may be positioned on or near a surface.

If a determination is made by the device orientation and movement logic230 that mobile device 100 is not in a moving vehicle, information fromany combination of the group of sensors may be combined to determine themobile device 100 orientation and whether the mobile device 100 may benear a surface. The device orientation and movement logic 230 providesan output, over the communication bus 135, to the surface detection andclassification logic 220 which includes the mobile device 100orientation, and a flag indicating that the mobile device 100 may be on,or close to, a surface. In response to the control signal, the surfacedetection and classification logic 220 proceeds with the method ofoperation shown in FIG. 4, which also occurs in operation block 205 ofFIG. 2. Therefore, FIG. 4 provides further details of operation block205. Thus, in FIG. 4 operation block 217, the surface detection andclassification logic 220 activates vibrator 117 while monitoring theaccelerometer 119 signal. That is, the accelerometer data is obtained asshown in operation block 219. The accelerometer 119 signal is thenanalyzed for characteristics such as frequency distribution, level, etc.and is compared to known predetermined signatures that are stored insystem memory 105. These signatures may be stored in the memory table106. In operation block 221, the known signatures based on theaccelerometer data are used to determine the surface type. The method ofoperation of FIG. 4 then ends and the surface detection andclassification logic 220 then provides an output, over the communicationbus 135, to the microphone configuration controller 210. The method ofoperation proceeds to operation block 207 in FIG. 2, further details ofwhich are provided in FIG. 5.

Therefore the microphone configuration controller 210 receivesinformation from the surface detection and classification logic 220regarding the surface type. The microphone configuration controller 210may also receive orientation information from the device orientation andmovement logic 230. The microphone configuration controller 210 thenuses the received information to configure a microphone array unit suchas the group of microphones 110, and determines which microphones to beactivated or turned off, the microphone gains at front end processing130, beamforming and signal conditioning based on a set of rules whichmay be stored in system memory 105. The rules are predetermined based onvarious surface types and signatures and orientations of the mobiledevice 100 with respect to such surface types. The microphoneconfiguration controller 210 may therefore control the beamformer 150 toform optimal virtual microphones for a given surface type and also sendscontrol signal 139 to the signal conditioner 160 to configure the noisesuppressor 165, and also the AGC 163 and de-reverberator 161 whenneeded.

These operations of the microphone configuration controller 210 occur inoperation block 207 of FIG. 2, further details of which are provided bythe flowchart of FIG. 5. Thus, in FIG. 5 operation block 223, themicrophone configuration controller 210 determines which microphones ofthe group of microphones 110 are available. For example, somemicrophones may be occluded or impeded by an absorptive surface, such asa couch cushion. Additionally, in some embodiments, some microphones maybe unavailable for voice recognition if they are being used by themobile device 100 for other audio activities, for example, audiorecording, voice communication, etc. However, such “in use” microphonesmay be used in some embodiments to form a virtual microphone and virtualvoice recognition channel using beamforming. Thus in operation block225, the microphone configuration controller 210 may turn somemicrophones on and others off accordingly. In operation block 227, themicrophone configuration controller 210 will determine whetherbeamforming should be applied and will send control signal 153 tobeamformer 150 to adjust parameters accordingly if applicable. That is,the microphone configuration controller 210 may operate changeoverswitch logic 140 to turn beamforming on or off accordingly for one ormore activated microphones, and configure the beamformer 150 by controlsignal 153. In operation block 229, the microphone configurationcontroller 210 will also determine what signal conditioning should beapplied given the mobile device 100 motion status, orientation andsurface type of any nearby surface. This determination may be made byperforming a lookup operation of signal conditioning configurations forknown conditions stored in system memory 105. In operation block 231,the microphone configuration controller 210 sends control signal 139 tothe signal conditioner 160 to configure signal conditioning accordingly.The method of operation of FIG. 5, and also of FIG. 2, then ends.

Returning briefly to operation block 201 and the flowchart of FIG. 3, ifa determination is made that the mobile device 100 is inside a movingvehicle, orientation and surface detection is attempted after accountingfor the variability in the accelerometer 119 and 3D gyroscope 115signals. The presence and characteristics of a surface is tested by thesurface detection and classification logic 220. The statistic of theinformation obtained, and criteria such as signal distribution,(variance, or other statistical measures) are used to determine if theattempted surface detection was successful or not. Upon a determinationthat the decision is un-reliable, a “default in-car configuration”signal is sent to the microphone configuration controller 210, otherwisethe microphones are configured accordingly for a detected surface asdescribed above.

The flowchart of FIG. 6 is useful for understanding an overall method ofoperation of the surface compensator 200. The method of operation beginsin operation block 301, and the surface compensator 200 obtains locationdata from the GPS logic 107 if location data is available. In operationblock 303, the surface compensator 200 obtains sensor data from any orall of the sensors such as, but not limited to, photosensor 109,photosensor 111, proximity sensor 113, 3D gyroscope 115, andaccelerometer 119. In decision block 305, the surface compensator 200checks the obtained location data, as well as gyroscope 115 andaccelerometer 119 data, to determine if the mobile device 100 is moving.If no movement is detected in decision block 305, then the method ofoperation proceeds to determine orientation of the mobile device 100 inoperation block 311.

However, if movement is detected in decision block 305, then the surfacecompensator 200 will proceed to decision block 307 and will determine ifthe mobile device 100 movement is caused by the mobile device 100 beinglocated in a vehicle. If not, then the method of operation againproceeds to operation block 311 to determine the mobile device 100orientation. However, if the mobile device 100 is determined to be in avehicle in decision block 307, then the surface compensator 200 mayadjust signal conditioning for the vehicle type in operation block 309.The surface compensator 200 may obtain settings for the signalconditioner 160 from system memory 105 based on known sensor datasignatures, that are known for in-vehicle acoustic conditions for agiven vehicle type, and may select settings that match the observedsensor signatures obtained in operation block 303.

In operation block 311, the surface compensator 200 determines theorientation of the mobile device 100 by reading the sensor data obtainedfrom 3D gyroscope 115. The orientation may indicate that there is alikelihood that the mobile device 100 is placed on or near a surface,such as, for example, when the mobile device 100 is laid on the table orsome other relatively flat surface. In response to such a determination,as shown in operation block 313, the surface compensator 200 may proceedto actuate surface sensors and obtain sensor data to attempt to detect anearby surface. For example, the surface compensator 200 may actuatevibrator 117 of mobile device 100 and monitor the response ofaccelerometer 119 to determine a surface signature and thus to determinewhether a surface is present. If a surface is detected in decision block315 then, in operation block 317, the surface compensator 200 maydetermine the surface type using the sensor data. For example, surfacecompensator 200 may access the memory table 106 stored in system memory105, to compare a measured signature to predetermined signatures thathave been previously measured for known surface types and for givenmobile device orientations. If no surface is detected in decision block315, method of operation ends as shown.

After detection of the surface in decision block 315, and determinationof surface type in operation block 317, the surface compensator 200 mayturn off microphones impeded by the detected surface and turn on anyavailable microphones as shown in operation block 319. In decision block321, surface compensator 200 may determine whether to apply beamforminggiven the orientation of the mobile device 100 and the detected surfacetype. If the surface compensator 200 determines that beamforming shouldnot be applied in decision block 321, then the surface compensator 200may proceed to select signal conditioning for the motion status,orientation and surface type as shown in operation block 325, afterwhich the method of operation ends as shown. If beamforming is to beapplied in decision block 321, then the surface compensator 200 mayselect a beamforming configuration based on the available microphonesand mobile device 100 orientation as shown in operation block 323. Forexample, the surface compensator 200 may access the memory table 106 orother memory tables stored in system memory 105 and obtain a beamformingconfiguration based on the detected surface type and mobile device 100orientation. An appropriate beamforming configuration may be selectedfrom a set of predetermined, stored beamforming configurations. Ifbeamforming is not to be applied in decision block 321, or if abeamforming configuration selection is completed in operation block 323,then the surface compensator 200 proceeds to operation block 325 andselects signal conditioning parameters appropriate for the motionstatus, orientation and surface type. The surface compensator 200 mayalso access system memory 105 to obtain signal conditioningconfigurations that are stored for predetermined known orientations ofthe mobile device and know surface types. The surface compensator 200then sends appropriate control signal 153 to the beam former 151 ifbeamforming is applied, and sends control signal 139 to signalconditioner 160 to accordingly adjust various parameters of thebeamformer 150 and signal conditioner 160, respectively.

Therefore, among other advantages of the various embodiments, microphoneconfiguration including application of beamforming and signalconditioning is adjusted by the surface compensator 200 to improve voicerecognition performance when a mobile device 100 is placed on, or near,an audio impeding surface whether reflective or absorptive. The surfacecompensator 200 determines, among other things, which microphones thatare not impeded or occluded for a detected mobile device 100orientation. The surface compensator 200 may utilize a single microphoneand may apply noise reduction algorithms suitable for the singlemicrophone. The surface compensator 200 may apply a table-top optimizednoise reduction algorithm and/or a table-top optimized beamformingconfiguration where pre-configuration information is stored in, andretrieved from, the memory table 106. The surface compensator 200 mayalso apply appropriately trained de-reverberation. The surfacecompensator 200 may use information from the proximity sensor 113 orfrom any of the group of sensors in various combinations to determineand select an optimal configuration for a microphone array such as thegroup of microphones 110 in response to detection of a nearby surfacewhich changes the mobile device 100 acoustic environment conditions.

Among other advantages, the embodiments also provide methods fordetermining the characteristics of the surface (such as when a mobiledevice is lying on a hard surface, or on a soft cushion such as couch orcar seat). In order to accomplish this task, a signal from the 3Daccelerometer 119 is examined by the surface compensator 200 both withand without external stimulus from the vibrator 117. Predeterminedcharacteristics of the signal may be stored in a database of signaturesobtained in different environments and stored in system memory 105. Forexample, the database of signatures may be stored in memory table 106 insome embodiments.

Another advantage of the embodiments is that actions are taken in orderto improve the signal sent to the voice command recognition logic 101,once accurate determination is made as to the presence of interferingsurfaces near-by the group of microphones 110. These actions fall intotwo categories; selecting which out of many available microphones areactivated, and how the signals from multiple microphones are combinedfor further processing.

For example, in some situations, such as where the mobile device 100 isdetermined by the surface compensator 200 to be laying upside-down on asoft cushion, the surface compensator 200 may further determine that aprimary microphone is occluded and can accordingly switch on a secondarymicrophone only (for example, a microphone located on the back of themobile device may be a “secondary” microphone) and configure the signalconditioner 160 to apply noise reduction for a single-microphone system.If multiple microphones are available, the appropriate set ofmicrophones can be identified by the surface compensator 200 andactivated. The surface compensator 200 may then further applybeamforming and signal conditioning configured appropriately for thesurface type and conditions.

Another advantage is that the presence of a very reflective surface canbe mitigated by configuring the use of appropriate microphones andturning on a de-reverberation processing, prior to noise reduction. Thetype of algorithm may be selected based on the severity of thecondition, or based on which microphone or microphones are active forthe condition observed. In situations, where accurate determinationabout the environment condition cannot be made, a safe defaultmicrophone configuration is used.

While various embodiments have been illustrated and described, it is tobe understood that the invention is not so limited. Numerousmodifications, changes, variations, substitutions and equivalents willoccur to those skilled in the art without departing from the scope ofthe present invention as defined by the appended claims.

What is claimed is:
 1. A method comprising: determining that a firstmicrophone, of a group of microphones of a mobile device, is impeded bya surface that is in proximity to the mobile device; and turning off thefirst microphone.
 2. The method of claim 1, further comprising: turningon a second microphone that is determined not to be impeded by thesurface.
 3. The method of claim 2, further comprising: detecting thesurface that is in proximity to the mobile device; and using sensor dataobtained from at least one sensor of the mobile device to determineacoustic reflectivity or acoustic absorptivity of the surface.
 4. Themethod of claim 3, wherein determining acoustic reflectivity or acousticabsorptivity of the surface comprises: actuating a vibrator of themobile device.
 5. The method of claim 1, further comprising: determiningorientation of the mobile device; determining that the first microphoneis impeded by the surface using the orientation and the firstmicrophone's known position in the mobile device.
 6. The method of claim3, further comprising: selecting signal conditioning parameters based onthe acoustic reflectivity or acoustic absorptivity of the surface. 7.The method of claim 6, wherein selecting signal conditioning parametersbased on the acoustic reflectivity or acoustic absorptivity of thesurface, comprises: selecting signal conditioning parameters from amemory table correlating a plurality of sets of signal conditioningparameters to surface types for predetermined conditions.
 8. The methodof claim 3, further comprising: applying de-reverberation to an outputof the second microphone when the surface is determined to bereflective.
 9. The method of claim 4, further comprising: turning on athird microphone that is determined not to be impeded by the surface;and beamforming the outputs of the second microphone and the thirdmicrophone to create a voice recognition virtual microphone.
 10. Themethod of claim 3, wherein detecting the surface that is in proximity tothe mobile device, comprises: actuating a vibrator of the mobile device;and obtaining data from an accelerometer of the mobile device where thedata exhibits a signature of the surface's surface type.
 11. Anapparatus comprising: a group of microphones; switch logic, operativelycoupled to the group of microphones, operative to switch each microphoneof the group of microphones either on or off in response to a firstcontrol signal; a signal conditioner, operatively coupled to the switchlogic and to the group of microphones, the signal conditioner operativeto receive outputs from each microphone of the group of microphones;voice recognition logic, operatively coupled to the signal conditioner;and a surface compensator, operatively coupled to the switch logic andto the signal conditioner, the surface compensator operative to:determine that a first microphone, of the group of microphones, isimpeded by a surface that is in proximity to the apparatus; and turn offthe first microphone by sending the first control signal to the switchlogic.
 12. The apparatus of claim 11, further comprising: a plurality ofsensors, operatively coupled to the surface compensator.
 13. Theapparatus of claim 12, further comprising: a vibrator, operativelycoupled to the surface compensator; an accelerometer operatively coupledto the surface compensator, wherein the accelerometer is one sensor ofthe plurality of sensors; and wherein the surface compensator isoperative to detect a surface in proximity of the apparatus by:actuating the vibrator; and obtaining data from an accelerometer wherethe data exhibits a signature of the surface's surface type.
 14. Theapparatus of claim 13, wherein the surface compensator is operative to:select signal conditioning parameters based on the signature of thesurface's surface type; and adjust the signal conditioner with theselected signal conditioning parameters.
 15. The apparatus of claim 11,further comprising: changeover switch logic, operatively coupled to theswitch logic; and a beamformer, operatively coupled to the changeoverswitch logic and to the signal conditioner, the beamformer operative toreceive and beamform microphone outputs from the changeover switch logicto create a virtual microphone and to provide at least one voicerecognition channel from the virtual microphone to the signalconditioner.
 16. The apparatus of claim 15, wherein the surfacecompensator is operatively coupled to changeover switch logic, andwherein the surface compensator is operative to send a second controlsignal to the changeover switch logic to control the changeover switchlogic to switch each microphone of the group of microphones either tothe beamformer or to the signal conditioner.
 17. The apparatus of claim12, further comprising: a 3D gyroscope wherein the 3D gyroscope is oneof the plurality of sensors; and wherein the surface compensator isoperative to: determine orientation of the apparatus by obtaining 3Dgyroscope data; and determine that the first microphone, of the group ofmicrophones, is impeded by the surface based on the orientation of theapparatus and the first microphone's know position in the apparatus. 18.The apparatus of claim 11, wherein the signal conditioner comprises anoise suppressor and a de-reverberator.
 19. The apparatus of claim 11,further comprising: global positioning system logic, operatively coupledto the surface compensator; and wherein the surface compensator isoperative to receive global positioning system data from the globalpositioning system logic and determine when the apparatus is in motionin a vehicle based on the global positioning system data.
 20. Theapparatus of claim 13, further comprising: non-volatile, non-transitorymemory, operatively coupled to the surface compensator; and wherein thesurface compensator is operative to: access the memory to compare theobserved signature of the surface's surface type to a table of storedpredetermined signatures and select the signal condition parameters fromthe table based on a matching predetermined signature.